Concept Drift Adaptation Methods under the Deep Learning Framework: A Literature Review

With the advent of the fourth industrial revolution, data-driven decision making has also become an integral part of decision making. At the same time, deep learning is one of the core technologies of the fourth industrial revolution that have become vital in decision making. However, in the era of epidemics and big data, the volume of data has increased dramatically while the sources have become progressively more complex, making data distribution highly susceptible to change. These situations can easily lead to concept drift, which directly affects the effectiveness of prediction models. How to cope with such complex situations and make timely and accurate decisions from multiple perspectives is a challenging research issue. To address this challenge, we summarize concept drift adaptation methods under the deep learning framework, which is beneficial to help decision makers make better decisions and analyze the causes of concept drift. First, we provide an overall introduction to concept drift, including the definition, causes, types, and process of concept drift adaptation methods under the deep learning framework. Second, we summarize concept drift adaptation methods in terms of discriminative learning, generative learning, hybrid learning, and others. For each aspect, we elaborate on the update modes, detection modes, and adaptation drift types of concept drift adaptation methods. In addition, we briefly describe the characteristics and application fields of deep learning algorithms using concept drift adaptation methods. Finally, we summarize common datasets and evaluation metrics and present future directions.

Platform Behavior under Market Shocks: A Simulation Framework and Reinforcement-Learning Based Study

We study the behavior of an economic platform (e.g., Amazon, Uber Eats, Instacart) under shocks, such as COVID-19 lockdowns, and the effect of different regulation considerations. To this end, we develop a multi-agent simulation environment of a platform economy in a multi-period setting where shocks may occur and disrupt the economy. Buyers and sellers are heterogeneous and modeled as economically-motivated agents, choosing whether or not to pay fees to access the platform. We use deep reinforcement learning to model the fee-setting and matching behavior of the platform, and consider two major types of regulation frameworks: (1) taxation policies and (2) platform fee restrictions. We offer a number of simulated experiments that cover different market settings and shed light on regulatory tradeoffs. Our results show that while many interventions are ineffective with a sophisticated platform actor, we identify a particular kind of regulation - fixing fees to the optimal, no-shock fees while still allowing a platform to choose how to match buyers and sellers - as holding promise for promoting the efficiency and resilience of the economic system. © 2023 ACM.

S-Nav: Safety-Aware IoT Navigation Tool for Avoiding COVID-19 Hotspots

In this article, we present a [Formula Omitted]-learning-enabled safe navigation system—S-Nav—that recommends routes in a road network by minimizing traveling through categorically demarcated COVID-19 hotspots. S-Nav takes the source and destination as inputs from the commuters and recommends a safe path for traveling. The S-Nav system dodges hotspots and ensures minimal passage through them in unavoidable situations. This feature of S-Nav reduces the commuter's risk of getting exposed to these contaminated zones and contracting the virus. To achieve this, we formulate the reward function for the reinforcement learning model by imposing zone-based penalties and demonstrate that S-Nav achieves convergence under all conditions. To ensure real-time results, we propose an Internet of Things (IoT)-based architecture by incorporating the cloud and fog computing paradigms. While the cloud is responsible for training on large road networks, the geographically aware fog nodes take the results from the cloud and retrain them based on smaller road networks. Through extensive implementation and experiments, we observe that S-Nav recommends reliable paths in near real time. In contrast to state-of-the-art techniques, S-Nav limits passage through red/orange zones to almost 2% and close to 100% through green zones. However, we observe 18% additional travel distances compared to precarious shortest paths.

A multi-agent reinforcement learning model for inventory transshipments under supply chain disruption

The COVID-19 pandemic has significantly disrupted global supply chains (SCs), emphasizing the importance of SC resilience, which refers to the ability of SCs to return to their original or more desirable state following disruptions. This study focuses on collaboration, a key component of SC resilience, and proposes a novel collaborative structure that incorporates a fictitious agent to manage inventory transshipment decisions between retailers in a centralized manner while maintaining the retailers' autonomy in ordering. The proposed collaborative structure offers the following advantages from SC resilience and operational perspectives: (1) it facilitates decision synchronization for enhanced collaboration among retailers, and (2) it allows retailers to collaborate without the need for information sharing, addressing the potential issue of information sharing reluctance. Additionally, this study employs non-stationary probability to capture the deeply uncertain nature of the ripple effect and the highly volatile customer demand caused by the pandemic. A new reinforcement learning (RL) algorithm is developed to handle non-stationary environments and to implement the proposed collaborative structure. Experimental results demonstrate that the proposed collaborative structure using the new RL algorithm achieves superior SC resilience compared with centralized inventory management systems with transshipment and decentralized inventory management systems without transshipment using traditional RL algorithms. [ FROM AUTHOR] Copyright of IISE Transactions is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full . (Copyright applies to all s.)

Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation

The spread of online misinformation threatens public health, democracy, and the broader society. While professional fact-checkers form the first line of defense by fact-checking popular false claims, they do not engage directly in conversations with misinformation spreaders. On the other hand, non-expert ordinary users act as eyes-on-the-ground who proactively counter misinformation - recent research has shown that 96% counter-misinformation responses are made by ordinary users. However, research also found that 2/3 times, these responses are rude and lack evidence. This work seeks to create a counter-misinformation response generation model to empower users to effectively correct misinformation. This objective is challenging due to the absence of datasets containing ground-truth of ideal counter-misinformation responses, and the lack of models that can generate responses backed by communication theories. In this work, we create two novel datasets of misinformation and counter-misinformation response pairs from in-the-wild social media and crowdsourcing from college-educated students. We annotate the collected data to distinguish poor from ideal responses that are factual, polite, and refute misinformation. We propose MisinfoCorrect, a reinforcement learning-based framework that learns to generate counter-misinformation responses for an input misinformation post. The model rewards the generator to increase the politeness, factuality, and refutation attitude while retaining text fluency and relevancy. Quantitative and qualitative evaluation shows that our model outperforms several baselines by generating high-quality counter-responses. This work illustrates the promise of generative text models for social good - here, to help create a safe and reliable information ecosystem. The code and data is accessible on https://github.com/claws-lab/MisinfoCorrect. © 2023 Owner/Author.

A Deep Learning-Based Preventive Measures of COVID-19 in a crowd using Reinforcement Model over GAN for Enhanced efficiency

The fast human-to-human spread of COVID-19 has caused significant lifestyle changes for many individuals. At the end of January 2020, the pandemic began, and many nations responded with varying degrees of testing, sanitation, lockdown, and quarantine centers. New normals of testing, sanitization, social separation, and lockdown are being implemented, and people are gradually returning to work and other daily routines. The COVID-19 infected population is monitored by testing individuals regularly. But it's a resource-heavy endeavor to test everyone without good reason. An optimum strategy is required to efficiently identify persons who are most likely to test positive for COVID-19. Sanitation is utilized for both persons and public spaces to eliminate germs. However, the disruption of governmental operations and economic development makes the use of lockdown and quarantine centers a resource-intensive endeavor. Conversely, it degrades the standard of living across a society. Furthermore, keeping people inside their houses or quarantine centers for an unlimited amount of time would not allow the government to care for everyone. These variables impact virus propagation, human health and happiness, available resources, and the economy's health, making their management resource-intensive. counting and density estimation are both attempts to create clever and efficient algorithms that can interpret the data provided by images to carry out Efficiency. GANs have been proven to have promising applications in overcoming the data dearth problem in COVID-19 lung image analysis. The Convolutional Neural Network (CNN) models built for the diagnosis of COVID-19 have benefited from the GAN-generated data used to refine their training. Moreover, GANs have helped improve the performance of CNNs by super-resolving pictures and performing segmentation. This work highlights the Reinforcement deep learning model over the fundamental constraints of the possible transformation of GANs-based approaches. This work proposes the model be developed with a new intelligent approach using RL to quantify these different types of testing considered for social distancing, face mask detection, limiting the gathering, and locking the location using the Q Learning technique. Different RL algorithms are implemented, and agents are equipped with these algorithms so that they may interact with the environment and learn the optimum method for doing so. © 2023 IEEE.

5GhNet: an intelligent QoE aware RAT selection framework for 5G-enabled healthcare network.

The COVID-19 outbreak has stimulated the digital transformation of antiquated healthcare system to a smart hospital, enabling the personalised and remote healthcare services. To augment the functionalities of these intelligent healthcare systems, 5G & B5G heterogeneous network has emerged as a robust and reliable solution. But the pivotal challenge for 5G & B5G connectivity solutions is to ensure flexible and agile service orchestration with acknowledged Quality of Experience (QoE). However, the existing radio access technology (RAT) selection strategies are incapacitated in terms of QoE provisioning and Quality of Service (QoS) maintenance. Therefore, an intelligent QoE aware RAT selection architecture based on software-defined wireless networking (SDWN) and edge computing has been proposed for 5G-enabled healthcare network. The proposed model leverages the principles of invalid action masking and multi-agent reinforcement learning to allow faster convergence to QoE optimised RAT selection policy. The analytical evaluation validates that the proposed scheme outperforms the other existing schemes in terms of enhancing personalised user-experience with efficient resource utilisation.

HRL4EC: Hierarchical reinforcement learning for multi-mode epidemic control.

Infectious diseases, such as Black Death, Spanish Flu, and COVID-19, have accompanied human history and threatened public health, resulting in enormous infections and even deaths among citizens. Because of their rapid development and huge impact, laying out interventions becomes one of the most critical paths for policymakers to respond to the epidemic. However, the existing studies mainly focus on epidemic control with a single intervention, which makes the epidemic control effectiveness severely compromised. In view of this, we propose a Hierarchical Reinforcement Learning decision framework for multi-mode Epidemic Control with multiple interventions called HRL4EC. We devise an epidemiological model, referred to as MID-SEIR, to describe multiple interventions' impact on transmission explicitly, and use it as the environment for HRL4EC. Besides, to address the complexity introduced by multiple interventions, this work transforms the multi-mode intervention decision problem into a multi-level control problem, and employs hierarchical reinforcement learning to find the optimal strategies. Finally, extensive experiments are conducted with real and simulated epidemic data to validate the effectiveness of our proposed method. We further analyze the experiment data in-depth, conclude a series of findings on epidemic intervention strategies, and make a visualization accordingly, which can provide heuristic support for policymakers' pandemic response.

Hierarchical Multi-agent Model for Reinforced Medical Resource Allocation with Imperfect Information

With the advent of the COVID-19 pandemic, the shortage in medical resources became increasingly more evident. Therefore, efficient strategies for medical resource allocation are urgently needed. However, conventional rule-based methods employed by public health experts have limited capability in dealing with the complex and dynamic pandemic-spreading situation. In addition, model-based optimization methods such as dynamic programming (DP) fail to work since we cannot obtain a precise model in real-world situations most of the time. Model-free reinforcement learning (RL) is a powerful tool for decision-making;however, three key challenges exist in solving this problem via RL: (1) complex situations and countless choices for decision-making in the real world;(2) imperfect information due to the latency of pandemic spreading;and (3) limitations on conducting experiments in the real world since we cannot set up pandemic outbreaks arbitrarily. In this article, we propose a hierarchical RL framework with several specially designed components. We design a decomposed action space with a corresponding training algorithm to deal with the countless choices, ensuring efficient and real-time strategies. We design a recurrent neural network-based framework to utilize the imperfect information obtained from the environment. We also design a multi-agent voting method, which modifies the decision-making process considering the randomness during model training and, thus, improves the performance. We build a pandemic-spreading simulator based on real-world data, serving as the experimental platform. We then conduct extensive experiments. The results show that our method outperforms all baselines, which reduces infections and deaths by 14.25% on average without the multi-agent voting method and up to 15.44% with it.

Same-day delivery with fair customer service

The demand for same-day delivery (SDD) has increased rapidly in the last few years and has particu-larly boomed during the COVID-19 pandemic. The fast growth is not without its challenge. In 2016, due to low concentrations of memberships and far distance from the depot, certain minority neighborhoods were excluded from receiving Amazon's SDD service, raising concerns about fairness. In this paper, we study the problem of offering fair SDD service to customers. The service area is partitioned into differ-ent regions. Over the course of a day, customers request for SDD service, and the timing of requests and delivery locations are not known in advance. The dispatcher dynamically assigns vehicles to make de-liveries to accepted customers before their delivery deadline. In addition to overall service rate ( utility ), we maximize the minimal regional service rate across all regions ( fairness ). We model the problem as a multi-objective Markov decision process and develop a deep Q-learning solution approach. We introduce a novel transformation of learning from rates to actual services, which creates a stable and efficient learn-ing process. Computational results demonstrate the effectiveness of our approach in alleviating unfairness both spatially and temporally in different customer geographies. We show this effectiveness is valid with different depot locations, providing businesses with an opportunity to achieve better fairness from any location. We also show that the proposed approach performs efficiently when serving heterogeneously wealthy districts in the city.(c) 2022 Elsevier B.V. All rights reserved.

Multi-objective route recommendation method based on Q-learning algorithm

With the continuous expansion of city scale and the advancement of transportation technology, route recommendations have become an increasingly common concern in academic and engineering circles. Research on route recommendation technology can significantly satisfy the travel demands of residents and city operations, thereby promoting the construction of smart cities and the development of intelligent transportation. However, most current route recommendation methods focus on generating a route satisfying a single objective attribute and fail to comprehensively consider other types of objective attributes or user preferences to generate personalized recommendation routes. This study proposes a multi-objective route recommendation method based on the reinforcement learning algorithm Q-learning, that comprehensively considers multiple objective attributes, such as travel time, safety risk, and COVID-19 risk, and generates recommended routes that satisfy the requirements of different scenarios by combining user preferences. Simultaneously, to address the problem that the Q-learning algorithm has low iteration efficiency and easily falls into the local optimum, this study introduces the dynamic exploration factor σ and initializes the value function in the road network construction process. The experimental results show that, when compared to other traditional route recommendation algorithms, the recommended path generated by the proposed algorithm has a lower path cost, and based on its unique Q-value table search mechanism, the proposed algorithm can generate the recommended route almost in real time. [ FROM AUTHOR] Copyright of Journal of Intelligent & Fuzzy Systems is the property of IOS Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full . (Copyright applies to all s.)

Enhancing Immersive User Experience Quality of StudoBot Telepresence Robots with Reinforcement Learning

The pandemic situation (Covid 19) brought new challenges in the education sector while simultaneously presenting unique opportunities for technology enabled services. The use of Mobile Robotic Telepresence systems in educational sector is promising as it provides means to significantly enhance the involvement and benefits to stakeholders involved in such interactions. An immersive user interaction with such a system depends on many aspects which are both static and dynamic. We approach the dynamic aspect of such interactions recognizing that the video and audio aspects of such a system will require fine tuning and adaptation. Closely related is the aspect of maintaining the necessary quality of network connection. Considering each of these aspects a reinforcement learning mechanism is incorporated to improve the overall user experience with such a system. A working system is built and experiments performed to demonstrate the effectiveness of the approach. Reward generation matrix, a crucial piece of data gathering from the environment, takes about 45 minutes, offline training time is less than a second, while the robot is able to cover the workspace in slightly less than a minute. The system is not limited to educational sector alone and provides a foundational framework to extend the concepts and principles to adjacent markets. © 2023 IEEE.

Computer-Aided System for COVID-19 Using Semi-supervised-based Ensemble Learning and Reinforcement Learning

Coronavirus Disease 2019(COVID-19) has shocked the world with its rapid spread and enormous threat to life and has continued up to the present. In this paper, a computer-aided system is proposed to detect infections and predict the disease progression of COVID-19. A high-quality CT scan database labeled with time-stamps and clinicopathologic variables is constructed to provide data support. To our knowledge, it is the only database with time relevance in the community. An object detection model is then trained to annotate infected regions. Using those regions, we detect the infections using a model with semi-supervised-based ensemble learning and predict the disease progression depending on reinforcement learning. We achieve an mAP of 0.92 for object detection. The accuracy for detecting infections is 98.46%, with a sensitivity of 97.68%, a specificity of 99.24%, and an AUC of 0.987. Significantly, the accuracy of predicting disease progression is 90.32% according to the timeline. It is a state-of-the-art result and can be used for clinical usage. © 2022 IEEE.

Patient–Robot Co-Navigation of Crowded Hospital Environments

Intelligent multi-purpose robotic assistants have the potential to assist nurses with a variety of non-critical tasks, such as object fetching, disinfecting areas, or supporting patient care. This paper focuses on enabling a multi-purpose robot to guide patients while walking. The proposed robotic framework aims at enabling a robot to learn how to navigate a crowded hospital environment while maintaining contact with the patient. Two deep reinforcement learning models are developed;the first model considers only dynamic obstacles (e.g., humans), while the second model considers static and dynamic obstacles in the environment. The models output the robot's velocity based on the following inputs;the patient's gait velocity, which is computed based on a leg detection method, spatial and temporal information from the environment, the humans in the scene, and the robot. The proposed models demonstrate promising results. Finally, the model that considers both static and dynamic obstacles is successfully deployed in the Gazebo simulation environment.

Evaluation of Masked Face Recognition of FaceNet Implemented with Machine Learning Algorithms

In image processing, Convolutional Neural Network (CNN) is an important tool for isolating image attributes for using with applications such as facial recognition. According to an outbreak of COVID-19, wearing masks has made face recognition less effective since face details are covered. FaceNet platform is a face feature extraction that is commonly applied to classification applications. Those applications embed FaceNet platform with supervised learning machine learning types to classify the considered objected on the detected image. Recently, Reinforcement Learning (RL) has been used in many applications on both prediction and classification tasks. However, the learning efficiency of RL has not been implemented and evaluated on masked face recognition yet. Therefore, the efficiency of the supervised learning techniques, ANN, KNN and SVM, are also implemented with the FaceNet platform for masked face recognition and they are compared with FaceNet platform implemented with the RL. The simulation results showed that ANN is the most efficient technique and followed by RL, KNN and SVM. The difference in efficiency (F1-scroce) between RL and the neural network was only 2%, but RL took four times more training time. © 2022 IEEE.

Reduce Cheating in e-Exams Using Machine Learning: State of the Art

Cheating on online exams becomes a black spot in distance learning environments. On the one hand, it threatens the credibility of these exams by violating the principle of equality and success on merit. On the other hand, it also has negative repercussions on the reputation of the institutions. Without a doubt, in the Covid-19 health crisis and following the recommendations of the World Health Organization to respect social distancing, the majority of establishments have adopted the distance learning system, including online exams. However, the difficulty of monitoring learner activity in remote settings characterizes this type of assessment by inequity. In practice, each establishment has relied on a monitoring solution adapted according to certain criteria in order to guarantee a fair passage of the exams and to control them well. AI-assisted proctoring tools add a layer of protection to online exams. In this article we will discuss and compare the different uses of Artificial Intelligence tools to reduce cheating in online exams, based on the use of Machine Learning techniques. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

DP-A*: For Path Planing of UGV and Contactless Delivery

The unmanned logistics and distribution urgently require a large number of unmanned ground vehicles(UGVs) under the influence of the potential spread of the Coronavirus Disease 2019 (COVID-19). The path planning of UGV relies excessively on SLAM map, and has no self-optimization and learning ability for the space containing a large number of unknown obstacles. In this paper, a new dynamic parameter-A* (DP-A*) algorithm is proposed, which is based on the A* algorithm and enables the UGV to continuously optimize the path while performing the same task repeatedly. First, the original evaluation functions of the A* algorithm are modified by Q-Learning to memory the coordinates of unknown obstacle. Then, Q-table is adopted as an auxiliary guidance for recording the characteristics of environmental changes and generating heuristic factor to overcome the shortcoming of the A* algorithm. At last, the DP-A* algorithm can realize path planning in the instantaneous changing environment, record the actual situation of obstacles encountered, and gradually optimize the path in the task that needs multiple explorations. By several simulations with different characteristics, it is shown that our algorithm outperforms Q-learning, Sarsa and A* according to the evaluation criteria such as convergence speed, memory systems consume, Optimization ability of path planning and dynamic learning ability. IEEE

Contact Tracing and Epidemic Intervention via Deep Reinforcement Learning

The recent outbreak of COVID-19 poses a serious threat to people's lives. Epidemic control strategies have also caused damage to the economy by cutting off humans' daily commute. In this article, we develop an Individual-based Reinforcement Learning Epidemic Control Agent (IDRLECA) to search for smart epidemic control strategies that can simultaneously minimize infections and the cost of mobility intervention. IDRLECA first hires an infection probability model to calculate the current infection probability of each individual. Then, the infection probabilities together with individuals' health status and movement information are fed to a novel GNN to estimate the spread of the virus through human contacts. The estimated risks are used to further support an RL agent to select individual-level epidemic-control actions. The training of IDRLECA is guided by a specially designed reward function considering both the cost of mobility intervention and the effectiveness of epidemic control. Moreover, we design a constraint for control-action selection that eases its difficulty and further improve exploring efficiency. Extensive experimental results demonstrate that IDRLECA can suppress infections at a very low level and retain more than 95% of human mobility. © 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Opportunity cost determines free-operant action initiation latency and predicts apathy.

BACKGROUND: Apathy, a disabling and poorly understood neuropsychiatric symptom, is characterised by impaired self-initiated behaviour. It has been hypothesised that the opportunity cost of time (OCT) may be a key computational variable linking self-initiated behaviour with motivational status. OCT represents the amount of reward which is foregone per second if no action is taken. Using a novel behavioural task and computational modelling, we investigated the relationship between OCT, self-initiation and apathy. We predicted that higher OCT would engender shorter action latencies, and that individuals with greater sensitivity to OCT would have higher behavioural apathy. METHODS: We modulated the OCT in a novel task called the 'Fisherman Game', Participants freely chose when to self-initiate actions to either collect rewards, or on occasion, to complete non-rewarding actions. We measured the relationship between action latencies, OCT and apathy for each participant across two independent non-clinical studies, one under laboratory conditions (n = 21) and one online (n = 90). 'Average-reward' reinforcement learning was used to model our data. We replicated our findings across both studies. RESULTS: We show that the latency of self-initiation is driven by changes in the OCT. Furthermore, we demonstrate, for the first time, that participants with higher apathy showed greater sensitivity to changes in OCT in younger adults. Our model shows that apathetic individuals experienced greatest change in subjective OCT during our task as a consequence of being more sensitive to rewards. CONCLUSIONS: Our results suggest that OCT is an important variable for determining free-operant action initiation and understanding apathy.

A reinforcement learning-based optimal control approach for managing an elective surgery backlog after pandemic disruption.

Contagious disease pandemics, such as COVID-19, can cause hospitals around the world to delay nonemergent elective surgeries, which results in a large surgery backlog. To develop an operational solution for providing patients timely surgical care with limited health care resources, this study proposes a stochastic control process-based method that helps hospitals make operational recovery plans to clear their surgery backlog and restore surgical activity safely. The elective surgery backlog recovery process is modeled by a general discrete-time queueing network system, which is formulated by a Markov decision process. A scheduling optimization algorithm based on the piecewise decaying [Formula: see text]-greedy reinforcement learning algorithm is proposed to make dynamic daily surgery scheduling plans considering newly arrived patients, waiting time and clinical urgency. The proposed method is tested through a set of simulated dataset, and implemented on an elective surgery backlog that built up in one large general hospital in China after the outbreak of COVID-19. The results show that, compared with the current policy, the proposed method can effectively and rapidly clear the surgery backlog caused by a pandemic while ensuring that all patients receive timely surgical care. These results encourage the wider adoption of the proposed method to manage surgery scheduling during all phases of a public health crisis.